Enhancements to improve the function of a biomimetic tactile sensor

ABSTRACT

Tactile sensors are disclosed that mimic the human fingertip and its touch receptors. The mechanical components are similar to a fingertip, with a rigid core surrounded by a weakly conductive fluid contained within an elastomeric skin. The deformable properties of the finger pad can be used as part of a transduction process. Multiple electrodes can be mounted on the surface of the rigid core and connected to impedance measuring circuitry within the core. External forces deform the fluid path around the electrodes, resulting in a distributed pattern of impedance changes containing information about those forces and the objects that applied them. Strategies are described for extracting features related to the mechanical inputs and using this information for reflexive grip control. Controlling grip force in a prosthetic having sensory feedback information is described. Techniques are described for enhancing the useful force range for impedance sensors by internally texturing the elastomeric skin.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation-in-part of U.S. patent applicationSer. No. 11/692,718, filed 28 Mar. 2007 (now U.S. Pat. No. 7,658,119,issued Feb. 9, 2010), and entitled “Biomimetic Tactile Sensor,” whichclaims priority to U.S. Provisional Patent Application No. 60/786,607,filed 28 Mar. 2006 and entitled “Biomimetic Tactile Sensor”; the entirecontents of both of which applications are incorporated herein byreference. This application is also a continuation-in-part of U.S.patent application Ser. No. 12/122,569 filed 16 May 2008, (now U.S. Pat.No. 7,878,075, issued Feb. 1, 2011), and entitled “Biomimetic TactileSensor for Control of Grip,” which claims priority to U.S. ProvisionalPatent Application No. 60/939,009, filed 18 May 2007 and entitled“Biomimetic Tactile Sensor for Control of Grip”; the entire contents ofboth of which applications are incorporated herein by reference. Thisapplication also claims the benefit of the following applications, allof which, and all references cited therein, are incorporated in theirentireties herein by reference: U.S. Provisional Patent Application No.61/041,861, filed 2 Apr. 2008 and entitled “Wearable Measurement Systemfor Shoulder Motion”; U.S. Provisional Patent Application No.61/041,865, filed 2 Apr. 2008 and entitled “Hand Motion CommandsInferred from Voluntary Shoulder Movement”; U.S. Provisional PatentApplication No. 61/041,867, filed 2 Apr. 2008 and entitled “Measurementof Sliding Friction-Induced Vibration for Tactile Feedback Control”;U.S. Provisional Patent Application No. 61/041,868, filed 2 Apr. 2008and entitled “Elastomer Patterning and Pressure Sensing Enhancements forFunctional Transduction in Electro-Hydraulic Impedance Sensing Devices”;and, U.S. Provisional Patent Application No. 61/042,182, filed 3 Apr.2008 and entitled “Spike-Like Regulator”.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH

This invention was made with government support under Grant No. EEC0310723, awarded by the National Science Foundation. The government hascertain rights in the invention.

BACKGROUND OF THE DISCLOSURE

1. Field of the Disclosure

This application relates generally to devices and methods to providetactile sensory information from robotic or prosthetic finger tipscomparable to the tactile sensing provided by human skin.

2. General Background and State of the Art

Present generations of robots lack most of the tactile sensorialabilities of humans. This limitation prevents industrial robots frombeing used to carry on delicate tasks of enormous practical relevance(such as assembly operations and handling of fragile objects) and, evenmore, it prevents the development of next-generation robots foroff-factory jobs (agriculture, home, assistance to the disabled, etc.).Future generations of robots will need to make use of a wide variety ofsensors and perceptual algorithms to identify and interact with objectsand surfaces in the external world, particularly in environment that areless structured than those in which industrial robots are used now.Taction, vision, and proximity are the sensory needs that, incombination or alone, are commonly accepted as desirable features ofrobots. Research on visual pattern recognition received considerableattention in recent years. Tactile recognition (the ability to recognizeobjects by manipulation) is an inherently active process. Unlike visualsensors (passive and located remotely from the object), tactile sensorscan be put in contact with the object to extract information about and,even more, such contact should be competently organized in order toextract the maximum degree of information from manipulative acts.

Humans who have suffered amputations of their hands and arms aregenerally provided with prosthetic limbs. Increasingly these prostheticsincorporate electromechanical actuators to operate articulations similarto biological joints, particularly to control the fingers to grasp andhold objects. Recent research has revealed how arrays of biologicaltactile receptors distributed throughout the soft tissues of the humanfinger tip are used normally by the nervous system to provide rapidadjustments of grip force when incipient slip is detected. Due tolimitations in currently available tactile sensing technology discussedbelow, currently available prosthetic fingers provide little or nosensing capabilities and cannot make use of these highly effectivebiological control strategies.

Engineered tactile sensors detecting mechanical stimuli can be groupedinto a number of different categories depending upon their construction.The most common groups are piezoresistive, piezoelectric, capacitive andelastoresistive structures. The common feature of all of these devicesis the transduction of mechanical strains or deformations intoelectrical signals. Tactile sensors are commonly used in the field ofrobotics and in particular with those robotic devices that pick up andplace objects in accordance with programmed instructions; the so-called“pick and place” class of robot. Unfortunately, while it would bedesirable for the above-listed groups of tactile sensors to respond inmuch the same way that the human finger does, many of them can provideonly limited information about contact with an object whose position,orientation and mechanical properties are highly predictable. Moregeneralized sensing requires a multiplicity of sensors and extensiveelectrical connections and signal processing circuitry. It is difficultto integrate these components into the tactile surfaces of manipulators,which are often required to have contoured, compliant surfaces tofacilitate handling of various objects. In order to achieve therequisite sensitivity, the individual sensors tend to be relativelyfragile and subject to mechanical damage over the wide dynamic range offorces to which they may be exposed. The large number of electricalconnections between sensors and signal processing circuitry tend to bedifficult and expensive to assemble, difficult to protect fromenvironmental hazards such as water and grit, and difficult orimpossible to repair if damaged.

A wide variety of technologies have been applied to solve the tactilesensing problem in robotics and medicine. Transduction mechanisms suchas optics, capacitance, piezoresistance, piezoelectricity, ultrasound,conductive polymers, etc. have all yielded viable solutions fordetecting either normal pressure distributions, shear forces, or dynamicfriction-induced vibrations but have required sensitive and fragiletransducers to reside close to the contact surface to accurately detectthese events. For example, most micro-electromechanical system (“MEMS”)sensors provide good resolution and sensitivity, but lack the robustnessfor many applications outside the laboratory.

Sensing of friction-induced vibrations has been a particular challengein the development of tactile sensors. These vibrations arise when acompliant sensor is stroked across a surface at some velocity. When thisoccurs, the power transferred into the skin by friction gives rise toacoustic vibrations in the skin and pulp of the finger. The biologicalfinger takes advantage of this phenomenon and has specialized sensors todetect these vibrations, which play an important role in slip-detectionfor reflexive grip-control. Many attempts to develop a sensor capable ofmeasuring such small vibrations have been made (Howe, Cutkosky, Dario),but they have required fragile dynamic sensors residing very close tothe contact surface to achieve the needed sensitivity. In this locationfragile sensing devices are at a high risk for damage and experienceshort lifetimes and expensive repair costs.

The curved, deformable nature of biological finger tips providesmechanical features that are important for the manipulation of the widevariety of objects encountered naturally. Many tactile sensing arrayshave been fabricated using MEMS but they are not suitable for mountingon such surfaces or for use in environments that include heavy loads,dust, fluids, sharp edges and wide temperature swings. If skin-likeelastic coverings are placed on top of sensor arrays, they generallydesensitize the sensors and function as low-pass temporal and spatialfilters with respect to incident stimuli, thereby attenuating dynamicinformation.

It is a general property of biological sensory receptors that they arehighly evolved structures in which the receptors themselves and thetissues in which they are located may contain many features designed toenhance their sensitivity and the quantity of information that they canprovide to the central nervous system. The skin contains multiple typesof mechanoreceptors to transduce a variety of mechanical events thatoccur during contact with physical objects. These receptors areconcentrated in sites such as the finger tips, where their sensitivityis enhanced by the mechanical properties of the skin, underlying pulpand bone, and adjacent fingernails.

The input-output properties of these biological transducers differgenerally from engineered transducers. Engineered transducers areusually designed to produce a linear response to a single mechanicalvariable such as normal or tangential force at a single point. Thesignals from arrays of such transducers can be combined according tosimple, analytical algorithms to extract orthogonal physical parametersof touch such as total force, center of force, directional force vectorand two-point resolution. Biological touch receptors are highlynonlinear and non-orthogonal. Their signals are combined by adaptiveneural networks to provide subconscious adjustment of motor output aswell as high level conscious perception associated with hapticidentification of objects. Neurophysiologists and psychologists oftencorrelate the activity of somatosensory receptors and design measures ofpsychophysical percepts according to canonical physical parameters, butthere is little evidence that the nervous system actually extractsdirect representations of such parameters as an intermediate stagebetween sensation and performance. In fact, information theory suggeststhat such an intermediate representation would add noise and reduceinformation content, which would place such a strategy at anevolutionary disadvantage.

Engineered sensors and their signal processing systems use linear,orthogonal representations because the downstream control systemsgenerally have been based on such inputs. This strategy may work wellfor engineered systems such as industrial robots that can performaccurately for highly constrained and predictable tasks. It is difficultto apply to anthropomorphic robots and prosthetic limbs that can performa broad and unpredictable range of tasks associated with activities ofdaily living. The problem may further be complicated by environmentalfactors in such environments (e.g. temperature, moisture, sharp edgesetc.), which tend to damage or bias sensitive and/or physically exposedtransducers.

U.S. Pat. No. 4,980,646, to Zemel (“Zemel”), is incorporated in itsentirety herein by reference and teaches a tactile sensor based onchanges in the local electrical resistance presented by a layer ofweakly conductive fluid whose shape is deformed by external forcesapplied to a deformable membrane. Zemel describes the application of avoltage gradient across the entire extent of the fluid by means ofelectrodes arranged on either side of the array of sensing strips, andthe measurement of the local strength of that gradient by differentialvoltage measurements between adjacent pairs of electrode strips. U.S.Pat. No. 4,555,953 to Dario et al., which is incorporated herein byreference in its entirety, teaches different techniques and materialsthat have been utilized for the construction of artificial skin-likesensors.

The following articles are referred to throughout the disclosure andtheir contents are incorporated by reference herein in their entireties:Lee M. H., Nichols H. R., Tactile sensing for mechatronics—a state ofthe art survey, Mechatronics 9:1-31 1999. Beccai L., Design andfabrication of a hybrid silicon three-axial force sensor forbiomechanical applications Sensors and Actuators, A. Physical. Vol.A120, no. 2: 370-382. 17 May 2005. Mei T., et al., An integrated MEMSthree-dimensional tactile sensor with large force range, Sensor andActuators 80:155-162, 2000. Beebe D., et al., A silicon force sensor forrobotics and medicine, Sensors and Actuators A 50:55-65,1995. Bloor D.,et al., A metal-polymer composite with unusual properties, Journal ofPhysics D: Applied Physics, 38: 2851-2860, 2005. Vasarhelyi G., et al.Effects of the elastic cover on tactile sensor arrays. Sensors andActuatorsl32:245-251, 2006. Helsel, M., et al., An impedance tomographictactile sensor, Sensor and Actuators. Vol. 14, No. 1, pp. 93-98.1988.Russell, R. A., Parkinson, S., Sensing surface shape by touch, IEEEInternational Conference on Robotics and Automation, Vol. 1 423-428,1993. Kenaly G., Cutkosky M., Electrorheological fluid-based roboticfingers with tactile sensing, Proceedings of IEEE InternationalConference on Robotics and Automation 1:132-136, 1989. Voyles R., etal., Design of a modular tactile sensor and actuator based on anelectrorheological gel, Proceedings of IEEE International Conference onRobotics and Automation, 1:132-136, 1989. Lee Y. K., et al., Mechanicalproperties of calcium phosphate based dental filling and regenerationmaterials, Journal of Oral Rehabilitation 30; 418-425, 2003. D. Merrill,et al., Electrical stimulation of excitable tissue: design ofefficacious and safe protocols, Journal of Neuroscience Methods, 141:171-198, 2005. A. Dalmia, et al., Electrochemical behavior of goldelectrodes modified with self-assembled monolayers with an acidic endgroup for selective detection of dopamine, Journal of Electrochemistry,430: 205-214, 1997. B. Piela, P. Wrona, Capacitance of the goldelectrode in 0.5 M sulfuric acid solution: AC impedance studies, Journalof Electrochemistry, 388: 69-79, 1994. Johansson R., et al.,Somatosensory control of precision grip during unpredictable pullingloads, Changes in load force amplitude, Experimental Brain Research 89:181-191, 1992. Birznieks I., et al, Encoding of direction of fingertipforces by human tactile afferents, Journal of Neuroscience.21:8222-8237, 2001. Flanagan J. R., et al. Control of fingertip forcesin multi-digit manipulation, Journal of Neurophysiology. 81:1706-1717,1999. Johansson R. S., Westling G., Roles of glabrous skin receptors andsensorimotor memory in automatic control of precision grip when liftingrougher or more slippery objects, Experimental Brain Research.56:550-564, 1984. Johansson R. S., Westling G., Signals in tactileafferents from the fingers eliciting adaptive motor responses duringprecision grip, Experimental Brain Research, 66:141-154, 1987. WestlingG., Johansson R. S., Responses in glabrous skin mechanoreceptors duringprecision grip in humans, Experimental Brain Research. 66:128-140, 1987.K. Hornik, et al., Multilayer feed forward networks are universalapproximators, Neural Networks, 2(5):359-366, 1989. Park, J. and I.Sandberg, Approximation and radial-basis-function networks, NeuralComputation 5, 305-316, 1993. Caudill, M.; Butler, C., UnderstandingNeural Networks: Computer Explorations; Volume 1: Basic Networks; TheMIT Press; Cambridge, Mass., 1992. D. Yamada, et al., Artificial FingerSkin having ridges and distributed tactile sensors used for grasp forcecontrol, Proc. IEEE/RSJ International Conference on Intelligent Robotsand Systems, pp. 686-691, 2001. Y. Mukaibo, et al., Development of atexture sensor emulating the tissue structure and perceptual mechanismof human fingers, Proc. of the 2005 IEEE International Conference onRobotics and Automation, pp. 2576-2581, 2005. Johansson R. S. andWestling G., Role of glabrous skin receptors and sensorimotor memory inautomatic control of precision grip when lifting rougher and moreslippery objects, Experimental Brain Research 56: 550-564, 1984. Cole K.J., Johansson R., Friction at the digit-object interface scales thesensory-motor transformation for grip responses to pulling loads,Experimental Brain Research, 95: 523-532, 1993. Johansson R., et al.,Somatosensory control of precision grip during unpredictable pullingloads, II Changes in load force rate, Experimental Brain Research 89:192-203, 1992. Gordon A., et al., Memory representation underlying motorcommands used during manipulation of common and novel objects, Journalof Neurophysiology 69: 1789-1796, 1993. Johansson R. S., Birznieks I.,First spikes in ensembles of human tactile afferents code complexspatial fingertip events, Nature Neuroscience 7:170-177, 2004.Butterfass, J., DLR-Hand II: Next generation of a dexterous robot hand,Proc. of the 2001 IEEE, International Conference on Robotics &Automation, Seoul, Korea, May 21-26, 2001. Mountcastle V. B., The viewfrom within: Pathways to the study of perception, The John HopkinsMedical Journal, 136:109-131, 1975. Wettels N., et al., BiomimeticTactile Sensor for Control of Grip, IEEE Rehabilitation Robotics, 2007,Proceedings of the IEEE International Conference on Robotics andAutomation, pp 109-114, 2001. N. Wettels, et al., “Biomimetic tactilesensor array” Advanced Robotics, vol. 22, no. 7, June 2008.

SUMMARY

Embodiments of the present disclosure are directed to biomimeticsensors, and related structures and processes. Exemplary embodiments ofthe present disclosure include sensory devices that have featurescomparable to features found in biological systems. In particular, theymay use biomimetic mechanical structures similar to those found in thefinger tip to endow a set of simple, robust electronic sensors with awide range of modalities and sensitivities similar to those found inbiological mechanoreceptors. Exemplary sensory devices include a sensorassembly whose basic form and function are similar to that of a humanfinger tip. The sensory device may have a biomimetic shape of a corewith covering skin and pulp (fluid reservoir) that results indistinctive and readily detectable patterns of impedance changes acrossan array of electrodes disposed on the core, to take advantage of thevarious distortions of the pulp produced by the contact parameters to bedetected and discriminated. High detection sensitivity and wide dynamicrange can be achieved for monitoring and/controlling the forces betweena manipulator and objects being manipulated. The biomimetic designs ofsuch sensor assemblies can allow for detection of stimulus features,e.g., by feature extraction circuitry, including those features that maybe most useful for automatic adjustment of contact force to achieve andmaintain stable and efficient grasp of an object. An exemplaryembodiment comprises a device through which a set of information isgenerated concerning tactile interaction between a manipulator and anobject to be manipulated and recognized. Such a device can beincorporated into autonomous robots, telerobots or prosthetic limbs. Thetactile information may be generated either by robot or prostheticfinger tips.

Biomimetic tactile sensors taught herein may possess softness,elasticity, and some mechanical resistance that mimics natural humanskin. Such sensors can detect and discriminate various aspects ofcontact with external objects, including the direction and magnitude offorce, the location, extent and shape of the contacting object, andsmall movements associated with impending slip. Furthermore, suchsensors may discriminate thermal properties of contacted objects throughheat-flow sensing.

Exemplary embodiments may employ a number of small, local electrodes,e.g., deployed in a curved array, as part of a sensing modality having ashape and mechanical properties to mimic those of a biological fingertip. Such electrodes can be used to detect changes in impedance of afluid within the sensor. Each sensing electrode may be energized toprovide an independent measure of the local mechanical deformations ofthe overlying membrane based on its impedance with respect to a remotecommon electrode. Further improvements are described to enhance thesensitivity and dynamic range of each sensing electrode by contouringthe inner surface of the overlying membrane. In further embodiments,neural networks may compute directly the actuator adjustments requiredto maintain stable grip of objects with a variety of shapes and forcevectors in a manner similar to that employed by neural control of thehuman hand.

Exemplary embodiments can include one or more temperature sensors, e.g.,thermocouple or thermistor, mounted to the surface of the core. Thesurrounding core is heated by the supporting electronics above ambient;when objects are contacted, they will cause temperature changes in thetemperature sensor (e.g., thermistor) consummate with their thermalproperties. These detected voltage changes can be exported to logic foranalysis.

Exemplary embodiments of biomimetic sensors and related techniques mayemploy a fluid pressure sensor either stand-alone or in addition toother types of sensor modalities, e.g., a number of small, localelectrodes deployed in a curved array to detect fluid impedance changes.Utilizing a fluid pressure sensor to detect the static and dynamicpressure of a fluid trapped between an elastomeric skin and a rigid coreallows for the detection of dynamic vibrations that arise as a device ofthis design is slid across a textured surface mimicking the function ofknown biological transducers. Furthermore the sensing of static pressurelevels of the fluid can be used to enhance the ability to extract thenormal forces as detected by impedance electrodes. Such fluid pressuresensing can confer a very high sensitivity to vibrations associated withdynamic friction-induced vibrations at the contact surface in a uniquelyrobust package.

To increase the robustness of the design the pressure sensor may belocated inside a rigid core where it is displaced from the contactregion of the finger. From this location it can be coupled with thefluid via a small pathway. A fluid with low compressibility and lowviscosity can be used to reduce the attenuation as the pressure wavestravel to the transducer in this remote location.

Embodiments can include a biomimetic tactile sensor that is sensitive tothe wide range of normal and shear forces encountered in robotic andprosthetic applications. Spatially resolved force distributions can bedetected through electrode impedance fluctuations as the conductivefluid profile deforms. A useful force range for biomimetic impedancesensors can be extended by internally texturing their elastomeric skin.One or more temperature sensors (e.g., a thermistor) can be placed onthe surface of a biomimetic to enable gross temperature sensing andthermal compensation of fluid conductivity for force sensing. Thetemperature sensors (thermistor) or the core material around it can beheated above ambient to detect heat flow to extract thermal features ofcontacted objects. Heat produced of dissipated by a microprocessorwithin control electronics can, for exemplary embodiments, be controlled(e.g., by appropriate software/firmware instructions), which is used toaffect thermal characterization of contacted objects.

One embodiment of the present device may consist of a set of sensorsthat work by measuring static and dynamic pressure in a fluid or gelthat is trapped by the elastomeric skin. The pressure transducer may bedeployed inside a substantially rigid core that is protected from directcontact with external objects. A feature of this design may be thelocation of mechanically vulnerable pressure transducer and signalprocessing circuitry, which can be wholly contained within thesubstantially rigid core. A related feature may be that this designenables methods of manufacture and repair that are simple and efficient.

The plurality of sensors and their associated mechanical structures havesimilarities to the biological relationships among the cutaneous neuralreceptors, the distal phalanx, overlying finger pulp and covering skinand nail. Information may be extracted from such a plurality of sensorswhereby such information can be related to canonical physicalrepresentations used to describe stimuli to be sensed, and/or used tocontrol automatic adjustments of grip forces similar to the neuralreflexes whereby humans maintain stable grip on complex objects.

One embodiment of present device may consist of a biomimetic tactilesensor that includes a dynamic pressure sensor to measure fluid pressurechanges due to sliding motion of the device skin over a surfaceencountered in robotic and prosthetic applications. In a preferredembodiment the fluid can have a low viscosity and low compressibilitysuch that these dynamic pressures are transmitted with littleattenuation through the fluid allowing for the pressure detectioncircuitry to be housed far away from the contact surface where it can beprotected from damage while still maintaining sensitivity to dynamictactile events. Another embodiment of the present device may consist ofa biomimetic tactile sensor capable of detecting vibrations relating tothe onset of slip. Another embodiment may consist of a biomimetictactile sensor with signal processing electronics for rapid extractionof spectral information of these vibrations for real-time slip detectionand automated grip control.

Another embodiment may consist of a biomimetic tactile sensor withsignal processing electronics which also adds normal and tangentialforce sensing to determine the true coefficient of friction between theinterface between the finger and the object. Another embodiment mayconsist of a method of controlling grip force based on the accuratemeasure of grip force as detected by the biomimetic tactile sensor andsignal processing electronics. Another embodiment may consist of amethod of controlling grip force that reduces power by relaxing gripforce until slip is detected and then increasing grip force to maintaingrasp. Another embodiment of the present device may consist of abiomimetic tactile sensor capable of detecting vibrations characteristicof surface texture. Another embodiment may consist of a biomimetictactile sensor with signal processing electronics for extraction ofspectral information related to texture for texture identification.Another preferred embodiment may consist of a biomimetic tactile sensorcapable of sensing normal and shear forces that are used in conjunctionwith an actuator to control the exploration of a surface in an optimalpattern to identify texture. Another embodiment may consist of abiomimetic tactile sensor capable of sensing friction-induced vibrationsin conjunction with a haptic display to replay this information on humanskin to take advantage of biological mechanisms of slip detection andtexture identification. Another embodiment may consist of a biomimetictactile sensor with external texturing similar to fingerprints or bumpswhich further enhances the response of the sensor to friction inducedvibrations.

It should be understood that other embodiments of biomimetic tactilesensor systems and methods according to the present disclosure willbecome readily apparent to those skilled in the art from the followingdetailed description, wherein exemplary embodiments are shown anddescribed by way of illustration. The biomimetic tactile sensor systemsand methods are capable of other and different embodiments, and detailsof such are capable of modification in various other respects.Accordingly, the drawings and detailed description are to be regarded asillustrative in nature and not as restrictive.

BRIEF DESCRIPTION OF THE DRAWINGS

Aspects of the disclosure may be more fully understood from thefollowing description when read together with the accompanying drawings,which are to be regarded as illustrative in nature, and not as limiting.The drawings are not necessarily to scale, emphasis instead being placedon the principles of the disclosure. In the drawings:

FIG. 1 depicts a longitudinal cross-section of a tactile sensor in theform of a finger pad, in accordance with exemplary embodiments of thepresent disclosure;

FIG. 2 depicts several views including a detail of a fill-portdisassembled (A) and assembled (B), and detail of resealing forcespresent on the fill-port (C), in accordance with exemplary embodimentsof the present disclosure;

FIG. 3 depicts a schematic of the electronic system for signal detectionand processing, in accordance with exemplary embodiments of the presentdisclosure;

FIG. 4 depicts forces diagrams of a tactile sensor, showing measurementof both normal and shear forces, at a cross section of the tactilesensor subjected to low shear force (A), and a cross section of thetactile sensor subjected to high shear force (B), in accordance withexemplary embodiments of the present disclosure;

FIG. 5 depicts an internally textured skin with asperities; aperspective view is shown (A), a side view depicts an uncompressedstate, (B) and a side view is shown depicting an applied force thatcompresses the textured rubber and narrows the flow path of the fluid tothe electrode surface (C), in accordance with exemplary embodiments ofthe present disclosure;

FIG. 6 illustrates a five-times-repeatability log-log plot of forceversus impedance for a skin incorporating inner-surface asperities withdimensions 0.25 tall×0.25 mm diameter, in accordance with exemplaryembodiments of the present disclosure;

FIG. 7 demonstrates pressure signals associated with contact and slidingfor a variety of contact events: tapping (A), pushing (B), sliding (C),and poking (D), along with an AC pressure signal (E) on the same timescale as normal and tangential forces as recorded by a force plate (F),in accordance with exemplary embodiments of the present disclosure;

FIG. 8 shows spectrograms of vibration signals used to discriminatetextured surfaces, in accordance with exemplary embodiments of thepresent disclosure;

FIG. 9 illustrates a cross section of synthetic skin with ridgesperforming the function of fingerprints to enhance vibrations, inaccordance with exemplary embodiments of the present disclosure;

FIG. 10 depicts a block diagram of a system with a manipulator, display,control features, and signal processing features, in accordance withexemplary embodiments of the present disclosure;

FIG. 11 depicts a flow chart for detecting friction coefficient, inaccordance with exemplary embodiments of the present disclosure;

FIG. 12 depicts a flow chart for controlling grip force with a knownfriction coefficient while also checking for slip, in accordance withexemplary embodiments of the present disclosure;

FIG. 13 depicts a flow chart for controlling grip force using only slipdetection, in accordance with exemplary embodiments of the presentdisclosure;

FIG. 14 depicts a flow chart for controlling exploratory movements fordiscriminating textures using vibration sensing, in accordance withexemplary embodiments of the present disclosure; and

FIG. 15 depicts a flow chart for controlling exploratory movements andsignal processing related to characterizing thermal properties ofcontacted objects, in accordance with exemplary embodiments of thepresent disclosure.

While certain embodiments depicted in the drawings, one skilled in theart will appreciate that the embodiments depicted are illustrative andthat variations of those shown, as well as other embodiments describedherein, may be envisioned and practiced within the scope of the presentdisclosure.

DETAILED DESCRIPTION

The detailed description set forth below is intended as a description ofexemplary embodiments of the tactile sensory system and method and isnot intended to represent the only embodiments in which the biomimetictactile sensor systems and methods can be practiced. The term“exemplary” used throughout this description means “serving as anexample, instance, or illustration,” and should not necessarily beconstrued as preferred or advantageous over other embodiments. Thedetailed description includes specific details for the purpose ofproviding a thorough understanding of the tactile sensory systems andmethods. It will be apparent, however, to those skilled in the art thatthe tactile sensory systems and methods may be practiced without thesespecific details. In some instances, well-known structures and devicesare shown in block diagram form in order to avoid obscuring the conceptsof the tactile sensory systems and methods.

A prosthetic hand or anthropomorphic robotic manipulator in accordancewith the present disclosure can combine several sensor assemblies at theends of appendages controlled by actuators, similar to the individualfinger tips of a biological hand or foot. Pad-like structures withsensors can be deployed on grip contact surfaces akin to the palmareminences over the heads of the metacarpal bones etc. One or more suchsensor assemblies could be built with various sizes and shapes andmounted in varying numbers and positions on a variety of hand-like orfoot-like locomotor supports to interact with external objects whileutilizing information derived from a plurality of contact sensors havingone or more types of sensing modalities.

The plurality of sensors and their associated mechanical structures canhave similarities to the biological relationships among the cutaneousneural receptors, the distal phalanx, overlying finger pulp and coveringskin and nail. Information may be extracted from such a plurality ofsensors whereby such information can be related to canonical physicalrepresentations used to describe stimuli to be sensed, and/or used tocontrol automatic adjustments of grip forces similar to the neuralreflexes whereby humans maintain stable grip on complex objects.

General Scheme

As described in detail below and in FIGS. 1-9, a sensor assembly 100 canconsist of mechanical elements corresponding to a biological fingertip,namely a molded, rigid core 1 corresponding to the bone, a molded,elastomeric skin 16 corresponding to the biological skin, a fluid-filledspace 2 corresponding to the deformable pulp of a biological finger, anda plate 22 affixing the skin 16 to the top surface of the fingertipcorresponding to a fingernail.

One embodiment of the present device may consist of a set of sensorsthat work by measuring electrical impedances through a weakly conductivefluid in contact with a plurality of electrodes. The electrodes may bedeployed on a substantially rigid core that is protected from directcontact with external objects by overlying deformable structures. Afeature of this design may be the location of mechanically vulnerableconnections between the electrodes and the signal processing circuitry,which are wholly contained within the substantially rigid core. Arelated feature may be that this design enables methods of manufactureand repair that are simple and efficient.

In exemplary embodiments of the present disclosure, three types ofsensors and their related signal processing circuitry may be utilized;such sensors may be incorporated alone or in various combinations:

i) Impedance Sensing—electrodes 6 on the surface of the core 1 that areused to detect changes in the impedance through the weakly conductivefluid that arise as a result of deformation of the overlying skin 16;

ii) Pressure Sensing—one or more pressure sensors 10 that detectpressure changes and vibrations conveyed through a fluidic path 4 fromthe skin 16 to the pressure sensors 10; and

iii) Temperature Sensing—a temperature sensor (e.g., thermistor 15)capable of detecting temperature and temperature fluctuations.

As described in FIGS. 10-15, one or more sensor assemblies 100 and theirassociated signal conditioning circuitry 9 can be incorporated intoactuated manipulators 500 that perform exploratory movements that bothrely on the signals from the sensors 100 to control their movements andinteractions with external objects and surfaces and utilize signals fromthe sensors 100 to identify various properties of external objects andsurfaces.

Mechanical and Fluidic Platform

Referring again to FIG. 1, a sensor assembly 100 consists of asubstantially rigid central core 1 covered by an elastomeric skin 16.The surface of core 1 contains a multiplicity of electrodes 6. A weaklyconductive fluid is introduced into space 2 thereby separating skin 16from that region of the core 1 where electrodes 6 are located. The edgeof skin 16 is shaped and dimensioned into collar 20 which provides aseal to core 1 to prevent leakage of the fluid; the force affecting thisseal may be augmented by shrinking band 21. The portion of skin 16 thatlies on the top portion of sensor assembly 100 is compressed againstcore 1 by plate 22, which is held in place by screws 23 tightened intothreaded inserts 24 in core 1. Thus the overall structure is similar toa human fingertip with a fingernail.

The weakly conductive fluid can be introduced into space 2 viahypodermic needle 26 during manufacture or servicing. Plates 19 moldedinto skin 16 create pressure seals around screws 23 and hole 25 wherethe pressurized region of skin 16 forms a resealable injection port whenpunctured by hypodermic needle 26. This process can be aided bymonitoring the fluidic pressure as sensed by a pressure sensor 10 toinflate to a prescribed hydraulic pressure level.

It is important for the reliable function of the sensors described belowthat there be no air bubbles in the weakly conductive fluid and that itbe contiguous with an inert, immiscible fluid contained within tube 4and inlet channel 12 of pressure sensor 10 contained within housing 11.One way to avoid air bubbles is to prevent their introduction in thefirst place. In an exemplary embodiment, this can be accomplishedthrough the following design features illustrated in FIG. 1 andfabrication steps:

-   1. Core 1 is designed to be injection molded around all of the    internal components illustrated in FIG. 1.-   2. At the time of molding, tube 4 and inlet channel 12 will have    already been filled with inert fluid and plugged with soluble keeper    3 made from a material that dissolved readily upon contact with the    weakly conductive fluid but not the inert fluid. Soluble keeper 3    contacts the walls of the mold so that it forms a continuous path    from the location of the resealable injection port to the space 2 to    be inflated with weakly conductive fluid.-   3. Core 1 can be inserted into skin 16 without introducing air    pockets by performing this step while submerged in a fluid.-   4. Soluble keeper 3 can be made from a material such as polyvinyl    alcohol that can be softened by heating and then displaced and    dissolved by the weakly conductive fluid when it is introduced via    hypodermic needle 26.

In one exemplary embodiment, the choice of the weakly conductive fluidis a solution of 0.4M NaI in propylene glycol. One choice of the inertfluid in contact with the pressure sensor is mineral oil. It isdesirable to protect the transductive electronic elements of pressuresensor 10 from contact with the salt ions that provide the electricalconductivity of the weakly conductive fluid, hence the requirement thatthese two fluids be immiscible in each other. It is also desirable theweakly conductive fluid not lose substantial volume by outward diffusionthrough skin 16 or gain substantial volume by inward diffusion of waterif the sensor is used in a humid or wet environment. This can beachieved by the selection of propylene glycol as the solvent base of theweakly conductive fluid and silicone elastomer for the skin 16. Theseexamples are illustrative only; other combinations of fluids andelastomers meeting these general requirements can be practiced withinthe scope of the present disclosure.

The components that are contained within core 1 can be preassembledmechanically and electrically via flex circuit 5 and loaded into theinjection mold used to form core 1. These mechanical and electricalconnections can be made by conventional techniques well-known topractitioners of the art, including soldering and conductive epoxy.These components include electrodes 6, which can be made from disks ofconductive, inert metals such as gold or platinum, thermistor 15,pressure sensor 10 in housing 11, and electronic components 9 asrequired for signal conditioning and processing as described later.

The sensor assembly 100 has been designed with many features such thatthe sensing mechanisms are robust and protected by the rigid core 1.This permits for the sensor assembly 100 to have a prolonged useful lifebefore needing replacement. The elastic skin 16 however through normaloperation will wear down at a much faster rate and will need to bereplaced in more frequent service intervals. The design of this tactilesensor consisting of a homogeneous elastic skin which contains noelectronics provides a unique advantage of low repair costs as thematerials for casting skins are comparatively cheap than skins used incompeting technologies.

In order to keep the costs of repairs low and maintain this competitiveadvantage a method to permit the end-user to replace skins as necessarywas implemented into the skin design. This was achieved with theaddition of a self-sealing skin 16 that once inserted onto the core 1produced compression forces into a groove in which it rests. Byintentionally creating an insertion hole of the skin that is smallerthan the groove in the core 1 it will rest in it is possible to producethe sealing properties needed to maintain the internal fluid pressureinside the tactile sensor. Care can also be taken such that the hole islarge enough that it will not tear as it passes over the tactile sensorbefore resting in the groove.

Referring to the detailed drawing in FIG. 2, injecting fluid into thedevice may be made possible by inserting a hypodermic needle 26 throughhole 25 in plate 22 which was designed to reseal after the hypodermicneedle 26 is removed. Novel aspects of this design include forming theelastic skin 16, rigid plate 22 and rigid core in such a way that oncecompressed and held together with screws 23 the resulting pressureproduces adequate sealing properties, this is illustrated in FIGS. 2 (A)and (B). At the location of the inflation hole 25 the skin 16 isthickened to permit for the development of compression forces from therigid surfaces it contacts once assembled. The core 1 and plate 22 aremade of a material of higher stiffness than the skin to ensure that theskin absorbs the deformation. As shown in FIG. 2 (C), the rigid surfaceson the core 1 and the place 22 that come into contact with thedeformable skin 16 are angled such that the resultant forces producecompression perpendicular to the axis in which the hypodermic needle 26will be inserted. Once the syringe is removed these forces act to resealthe tiny hole that was created when the hypodermic needle 26 wasinserted. This allows for the skin 16 to have a longer lifetime overmore inflation cycles before needing to be replaced.

The following references, which are incorporated by reference in theirentireties, teach various features that may be utilized in the presenttactile sensor devices and methods: U.S. Pat. No. 6,871,395 to Scher etal. teaches connecting electrically conductive elastomer to electronicsand U.S. Pat. No. 6,529,122 to Magnussen et al. teaches measuringcontact resistance between work pieces, U.S. Pat. No. 5,905,430 toYoshino et al. for detecting state of contact between a contact memberand a work piece, U.S. Pat. No. 5,033,291 to Podoloff et al. forflexible tactile sensor for measuring foot pressure distributions; U.S.Pat. No. 5,014,224 to Hans for determining location and amount ofexerted pressure; U.S. Pat. No. 4,817,440 to Curtin for identifyingcontact force and the contact pattern; U.S. Pat. No. 4,526,043 to Boieet al. for conformable tactile sensor; and U.S. Pat. No. 4,481,815 toOverton for determining a parameter of an object being contacted by thetactile sensor.

Sensing Elements and Signal Conditioning

Impedance Sensing may be accomplished by measuring changes in theelectrical impedance among electrodes 6 whose distribution and locationon the contoured surface of the core 1 may be a key factor in thesensing properties of the sensor assembly 100. One embodiment ofdetection circuitry 9 is illustrated schematically in FIG. 3 anddescribed in more detail below. The electrical impedances Z₀₋₁₅ someasured can be dominated by the dimensions and electrical properties ofthe weakly conductive fluid in space 2. These dimensions are changed byforces applied to skin 16, which result in deformation of the skin 16and displacement of the weakly conductive fluid. Multiplexer 215 selectseach electrode in turn for connection to the measurement circuitry underthe control of microcontroller 230. The impedance Z of a selectedelectrode 6 can be measured by applying a voltage to a circuitconsisting of one or more excitation electrodes 6 located elsewhere onthe surface of core 1 in series with selected electrode 6 and theintervening weakly conductive fluid path between them plus a fixedreference resistor 220 labeled R_(load). Advantageously, the appliedvoltage is an alternating or pulsatile voltage that does not result innet direct current flow through the electrodes 6 which would tend todamage them by inducing electrolysis and corrosion. In the preferredembodiment in FIG. 3, this is achieved by using a clock signal CLK inseries with a DC blocking capacitor, but other configurations would beobvious to someone normally skilled in the art. At the peak of eachpulse in clock signal CLK, the voltage across R_(load) is measured byanalog-to-digital convertor contained within microcontroller 230.R_(load) plus the selected one of Z₀₋₁₅ constitute a voltage dividersuch that the voltage measured across R_(load) varies inversely with theselected impedance from Z₀₋₁₅.

Pressure Sensing may be accomplished by one or more pressure sensors 10,whose transductive elements are represented by the variable resistors inthe bridge circuits P_(AC) and P_(DC) in FIG. 2. Pressure Sensing can beusefully divided into the relatively large, quasistatic pressures P_(DC)in the fluids, which are measured by pressure sensing subsystem 205, andthe much smaller, audio frequency vibrations P_(AC) arising as the skin16 slides over objects in contact with sensor assembly 100, which aremeasured by vibration sensing subsystem 205. The sensitivity and dynamicrange of a pressure sensor 10 depends on the nature of the materialcontained within its associated reference channel 13 within its housing11. In the preferred embodiment, pressure sensing subsystem 205 includesa wide dynamic range pressure sensor 10 and its reference channel 13 isfilled with an air pocket that is trapped there by cap 14. The vibrationsensing subsystem 205 includes a narrow dynamic range pressure sensor 10that needs only to be sensitive to alternating fluctuations in pressure.This is achieved by trapping an air pocket in its reference channel 13but connecting the other side of the reference channel via a small gagetube (not illustrated) back to its inlet channel 12.

The combination of an incompressible fluid path in the inlet channel 12and a compressible air bubble in series with the high fluidic resistanceof the small gage tube constitutes a mechanical high pass filter,protecting pressure sensor 10 from large hydrostatic pressures that mayarise in the fluid during the initial inflation of space 2 or firmcontact with external objects. This allows for use of a more sensitivetransducer with smaller dynamic range but larger signal-to-noise ratio.Alternatively, if a single pressure sensor 10 is available withsufficient dynamic range and signal-to-noise values, both pressure andvibration information may be extracted from it using analog or digitalsignal conditioning means as would be obvious to one normally skilled inthe art. The signals from the one or more pressure sensors 10 aresuitably amplified by X100 and digitized by ADC channels inmicrocontroller 230.

Temperature Sensing may be accomplished via thermistor 15 and associatedcircuitry described in FIG. 3. Temperature Sensing may be usefullydivided into i) quasistatic sensing of ambient temperature T_(DC) bythermistor 15 in DC temperature subsystem 240; and ii) dynamic sensingof local fluctuations in temperature T_(AC) by AC temperature subsystem235. Such fluctuations in temperature that may be induced by contactbetween the region of the skin 16 overlying thermistor 15 (asillustrated mechanically in FIG. 1) and an external object at atemperature different from the ambient temperature. The ambienttemperature in sensor assembly 100 depends on the equilibrium betweenheat energy generated through the operation of all of its electroniccomponents 9 and the heat energy conducted away by various conductive,convective, and radiative losses. The heat energy generated will tend tobe dominated by the contribution from microcontroller 230, whose powerdissipation can be controlled dynamically by its software, for exampleby changing its clock rate or sleep intervals. The ambient temperaturein the environment around sensor assembly 100 can thus be inferred bythe amount of heat energy that can be generated to keep thermistor 15 ata desired temperature, such as the 37 C core temperature of the humanbody. In a preferred embodiment, T_(DC) is sensed by a voltage dividerand amplifier, one of many measurement circuits that would be well-knownto practitioners of the art. The fluctuations T_(AC) will depend on thedifference between the temperature of the core 1 and the temperature ofa contacting object, the thermal conductivity of that contacting object,and the location, extent and force of contact between sensor assembly100 and the contacting object. These fluctuations can be high-passfiltered and amplified to generate T_(AC). The signals from DCtemperature subsystem 240 and AC temperature subsystem 235 are digitizedby ADC channels in microcontroller 230.

Signal Processing and Transmission

It is useful to minimize the number of electrical connections that canbe made from the control system for a manipulator 500 (described in moredetail below in FIG. 10) to the one or more sensor assemblies 100incorporated into a manipulator 500. This can be done by multiplexingthe data derived from all of the sensing functions in each sensorassembly 100 into a serial stream of digital bits. The circuitryrequired for energizing the sensors, analog signal conditioning,digitization and serialization into a standard protocol (e.g., SPI orI2C) may be located physically in the fingertip, along with the variouselectrodes 6, other transducers and their electrical connections. In anexemplary embodiment a simple circuit can be built from off-the-shelfcomponents, including integrated circuits that could be procured assurface-mount packages or bare dies and incorporated onto flex circuit5.

Representation of Features of Contact by Impedance Sensors

FIG. 4 illustrates a cross-sectional view of sensor assembly 100 that isorthogonal to the view in FIG. 1. The positioning of the electrodes 6with respect to the contours of the core 1 and overlying fluid-filledspace 2 and skin 16 may cause distinct patterns of change in the variousimpedances Z₀₋₁₅ measured by electronic components 9 as the sensorassembly 100 contacts various objects and surfaces with various forcevectors. It may be useful to identify how different aspects of anyparticular stimulus parameter to be sensed will influence the array ofelectrodes comprising the sensor assembly 100.

In the example illustrated in FIG. 4, a single point of contact mayexperience various combinations of normal and tangential forcecomponents (labeled Fnorm and Ftang, respectively), which result indistributed changes in the impedances measured at each of the electrodes6 as a result of sliding and deformation of skin 16 with respect to core1 and plate 22. In this example, it is advantageous for the fluid inspace 2 to lubricate the interface between the inner surface of skin 16and the surface of core 1 and electrodes 6 in order to facilitatesliding between them. One choice for such a fluid that is compatiblewith the other requirements upon this fluid is propylene glycol. Pendingrelated U.S. patent application Ser. No. 11/692,718, incorporated hereinby reference in its entirety, describes in detail various mechanismswhereby various features of contact can be discriminated, includingcontact force, centroid and area of force, eccentricity of force, shapeof the external object, vector of force, and object hardness andsoftness.

If different features of contact between sensor assembly 100 andexternal objects result in sufficiently distinct output patterns acrossall of the elements of the sensor, then some information (particularlyabout position such as force centroids and areas) could be extractedanalytically, based on a reasonable mathematical model. Sensor assembly100 can have properties similar to the biological fingertip, however, soit may likely require non-analytical signal processing methods similarto those employed by the biological nervous system.

The temporospatial distribution of activity in the biological touchsensors depends complexly on the inherent sensitivity of the sensors,their distributions throughout the tissues of the fingertip and theforces that the fingers apply to external object, as well as on thenature of the external object itself. Similarly, in electrode array 6 oftactile sensors, force magnitude and location interact with each other.For example, the same force vector applied close to the nail bed maycreate a different amount of net impedance change than if applied to thefingertip; the total change in impedance may not be used as a measure ofthe applied force unless corrected for the position. At higher forcelevels the information about position may be blurred because ofnonlinear changes in electrode impedance as the inside surface of theskin makes contact with the electrodes. This is similar to thesaturation of light touch receptors and the need to incorporateinformation from deep touch and nociceptors in biological skin.

The characterization experiments described above may produce a rich dataset consisting of pairs of input vectors (describing location andcomponents of applied force) and output vectors (voltages related toimpedances of the electrode array). These may be used to train neuralnetworks for various tasks. This approach can be used to determine thediscriminability of various input conditions or, conversely, todetermine the ability to generalize a single parameter such as magnitudeof forces applied to different portions of the finger tip. For the forceintensity extraction, a multi-layer perceptron (MLP) and radial-basisneural network may be used initially because both have proven to be ableto approximate any given non-linear relation when a sufficient number ofneurons are provided in the hidden layer. Two-point discrimination maylikely be possible but may depend critically on the thickness andviscoelastic properties of the skin. It may be feasible to employalgorithms known as neural networks that may function similar to thoseembodied in the nervous system in order to identify the nature of thecontact state in terms of feature of contacted objects andspatiotemporal distribution of contact forces. That is, neural networkscan be trained by learning to respond in a useful manner to thosefeatures of any stimulus that can be discriminated, as would be obviousto one normally skilled in the art. Active feature extraction isdescribed in more detail in FIGS. 10-15 and related text.

Enhancement of Dynamic Range of Force Sensing

The dynamic range of forces that can be measured usefully by ImpedanceSensing depends on the rate at which space 2 over a given electrode 6tends to be occluded by increasing force applied to the overlying skin16. The impedance of a fluid channel depends upon the resistivity of thefluid, the length of the channel and its cross-sectional area: R=(ρL)/A.All else being equal, a channel with a larger average cross-sectionalarea will have lower impedance.

If both the core and the skin are smooth, there may be a tendency forthe measured electrical impedance of the sensing electrode to riseabruptly and to saturate when the skin is pressed against it, forming atight seal. By controlling the size, distribution and mechanicalstiffness of surface textural features, the useful dynamic range ofsensing can be greatly extended.

As illustrated in FIG. 5, the inner surface of skin 16 can includeasperities 18 molded into the contour of the skin, for example, byforming skin 16 in an injection mold whose corresponding surface is anegative of the desired pattern of asperities 18. Other methods offabrication of textures would be obvious to one normally skilled in theart, including photolithography, incorporation of soluble particles,plasma etching, etc. Related U.S. patent application Ser. No. 11/692,718teaches that it may be desirable to have the inside surface of the skinpatterned with “bumps and/or ridges”.

FIG. 5A illustrates one advantageous embodiment of a useful surfacetextural feature consisting of a repeated pattern of closely spacedasperities 18, each of which is a pyramidal or columnar protrusions fromthe inner surface of skin 16. Such a pattern tends to leave channels ofconductive fluid on the surface of core 1 and electrodes 6, whichchannels may be gradually compressed and narrowed with increasingcompressive force applied to the skin, as illustrated in cross-sectionalviews in FIGS. 5B and 5C. FIG. 6 illustrates a wide dynamic range ofinput normal forces (log scale on abscissa) that result in a widedynamic range of output electrode impedance (log scale on ordinate) fora pattern of cylindrical asperities 0.25 mm tall, 0.25 mm in diameterand 0.5 mm apart in a regular, square grid. The dynamic range can beextended by texturing the inner surface of the elastomeric skin.

When the skin contacts the core and occludes a given electrode, a sealis formed, isolating it from the fluid while impedance increases toinfinity. This happens at low force levels, causing the device tofunction as a switch rather than a usable transducer. Texturing theinternal surface of the skin allows for fluid pathways to exist evenafter the internal surface of the skin has been compressed against theelectrode.

Other Aspects of Manufacture

A suitable sensor core can be fabricated by creating a negative mold ofthe desired core shape, advantageously using a relatively soft materialsuch as machinist's wax. Components that need to be present on thesurface of the mold (such as the electrode contacts and the capillaryfill tube opening) can be affixed in the desired locations by pressingthem onto the surface of the mold. Any desired mechanical or electricalconnections from those components can be made to electronic circuits orconnector pins in the open mold. All of the components and theirinterconnections are then embedded in the core material that is pouredinto the mold around the components and cured in place. High densitypolyurethane can be used to form the core. This method lends itself wellto resealing the tactile array for different applications, changing thecurvature of its surface, and/or changing the number and distribution ofelectrode contacts.

As described above, it may be advantageous and feasible to incorporatemost or the entire signal conditioning circuitry and connections to theelectrodes within the fingertip itself. This may greatly reduce thenumber of electrical connections that may be made to transmit the datafrom the tactile sensor array to whatever controller requires thosedata. The above-described method of forming the core by pouring andpolymerizing the core material may be particularly well-suited forcreating a rugged protective enclosure around such signal conditioningcircuitry, which may obviate the need for bulky and expensive hermeticpackaging and feedthroughs for the electronic circuitry. The materialchosen for the core should be relatively impermeable to the fluid chosento inflate the fingertip.

Dynamic Sensing of Fluid Pressure

FIG. 7 illustrates the signals produced by a pressure sensor 10connected to the incompressible fluid occupying space 2 of a sensorassembly 31 as it contacts a flat, hard surface with different patternsof force and motion. In FIG. 7A-D, the output of pressure sensingsubsystem 210 reflecting P_(DC) is provided as a function of time duringvarious contact events. In FIG. 7E, the output of vibration sensingsubsystem 205 reflecting P_(AC) is provided as a function of time, alongwith the normal and tangential reaction forces (FIG. 7F) recorded by acommercial force-plate under the surface being contacted.Friction-induced vibrations in skin 16 are sensed as fluctuations influid-pressure by pressure sensor 10 having a bandwidth of approximately0-1000 Hz. An exemplary embodiment included a commercially availablepressure transducer (Honeywell Model #40PC015G1A) and based on thevibration sensitivity of the human finger an analog first-order low-passfilter with center frequency at 1000 Hz and a digital sampling rate of2500 samples/sec were used.

Fidelity of transmission of acoustic signals is a concern when measuringdynamic fluctuations in pressure from a signal source at a remotelocation. Important parameters to consider are the geometries ofpathways involved between the signal source and recording site as wellas the wavelengths of signals being recorded. The wavelength of a signal(λ) can be calculated from the speed of sound in a media (c) and thefrequency of the signal (f) from the following formula:

$\lambda = {\frac{c}{f}.}$Given the inherently long wavelengths of these frequencies in candidatefluids, even the shortest wavelength of interest is on the order of 1meter. This makes it possible to displace the pressure sensor 10 shortdistances in with respect to this wavelength without having to worryabout acoustic interference due to the geometry of fluidic pathways.This is desirable because it removes the pressure sensor 10 from thecontact surfaces of the skin 16 where it may be damaged from impact.Additionally, moving the pressure sensor 10 only a few centimeters fromthe contact surface and connecting it to the fluid through a tube 4enables pressure sensor 10 to be housed safely within the core 1 withnegligible losses from acoustic interference. Selection of anincompressible fluid with a low viscosity such a propylene glycolfurther enhances sensitivity.

Slip Detection from Friction-Induced Vibrations

As illustrated in FIG. 7, pressure sensor 10 provides easily detectedaudio frequency signals when skin 16 starts to slip over a contactobject. Dynamic information correlated with slip has been shown to liein frequencies between 50-700 Hz using this approach. This correlateswell with what is known of biological slip detection mechanisms.Presence of frequency content within this band is not unique to slip andis also common to other dynamic events such as shock or contact.However, slip events are unique in that they have significantly lessspectral information in low frequency bands (0-50 Hz) with respect tothese other dynamic events, as shown in FIGS. 7 (A), (B) and (D). Inexperiments conducted by the inventors, the absence of low-frequencyvibrations combined with the presence of high-frequency vibrationstended to occur only during slip when contacting passive objects.

Texture Discrimination from Friction-Induced Vibrations

As noted in FIG. 8, the sensor detects different spectral patterns as itis slid across surfaces of different texture. Thus the sensor is capableof performing texture discrimination tasks when used in conjunction withmechanical means to slide the sensor over textures to be discriminated.Textures have been shown to provide characteristic, repeatable, andidentifiable frequency patterns when observed visually on short-timeFourier transform spectrograms or when listened to acoustically. Onesuch signal processing method for extracting this in an automatedfashion is to use a short-time Fourier transform with a wide timewindow. This wide time window is desired to improve the frequencyresolution of this transform and improve the ability to discriminatebetween different textures.

External Texturing to Enhance Friction-Induced Vibrations

To enhance vibration sensing, a regular pattern of ridges 17 similar tothose found on human fingers or bumps similar to those found on thefingerpads of raccoons can be incorporated onto the exterior of theelastic skin as depicted in FIG. 9. This would permit for theenhancement of slip detection as well as improve the spectral contentused for the determination of texture. This is accomplished due to theinherent structure of elastic structures in this shape. As illustratedin FIGS. 9 (A) and (B), the cantilever structure of the elastic ridges17 allows for energy to be stored in the bending of these structures astangential movements are applied. This stored elastic potential producesa force at the contact surface which is maintained by the frictionbetween the ridges 17 and the object being contacted.

As depicted in FIG. 9 (C), when this force exceeds the limits of whatcan be applied with friction, the ridged structure snaps loose andreleases this stored energy which manifests as vibrations in the skin16. Using mechanisms as described above, these vibrations in the skin 16are conveyed through the incompressible fluid occupying space 2 anddetected by the pressure sensor 10. In the preferred embodiment, theseridges 17 are incorporated into the surface of skin 16 during itsmanufacture by injection molding into a cavity whose surface includes anegative of the desired ridge pattern. Other methods of forming orattaching ridges 17 or other textural patterns performing a similarfunction would be obvious to one normally skilled in the art.

Multimodal Signal Processing

It is highly advantageous to combine the three types of sensorsdescribed above because the interpretation of signals from one type ofsensor may depend on conditions that can be sensed by another type ofsensor. The following examples are illustrative but are not intended tobe exhaustive.

Impedance Sensing depends on the conductivity of the weakly conductivefluid, which tends to vary directly with the temperature of the fluid.The ambient core temperature detected as T_(DC) can be measured andstabilized via a feedback control algorithm that adjusts the powerdissipation of electronic components 9.

Pressure Sensing of vibrations produced by slippage between sensorassembly 100 and the contacting object depend on the location, extentand force of contact between them. These mechanical factors can all beextracted from the Impedance Sensing and used in a feedback controlscheme to adjust the nature of the contact to standardized values, muchas humans tend to standardize their exploratory movements when trying toidentify the texture of an unknown surface.

Temperature Sensing of fluctuations produced by contact with an externalobject depend on the location, extent and force of contact betweensensor assembly 100 and the contacting object. These mechanical factorscan all be extracted from the Impedance Sensing and used in a feedbackcontrol scheme to adjust the nature of the contact to standardizevalues, much as humans tend to standardize their exploratory movementswhen trying to identify an unknown object.

FIG. 10 provides a block diagram of a complete system 1000 for controland sensing in a multi-articulated manipulator, in accordance with anexemplary embodiment of the present disclosure. Such a manipulator caninclude one or more sensor assemblies 100 and one or more poweredactuators 600 responsive to a controller 400. Sensor signals fromconditioning circuitry 9 in each sensor assembly 100 is transmitted tofeature extraction means 300 and may be used to provide sensory feedbackto controller 400 according to the requirements of various applicationsdescribed below. In some applications, the functions performed bycontroller 400 are partially or wholly under the command of an externaloperator 900 which may be a human operator or other source of commandssuch as would be generated in an autonomous robot.

In the case of a human operator 900, sensory information from featureextractor 300 may be advantageously provided to operator 900 by one ormore of the many haptic display interfaces now available or underdevelopment, including tactors for generating force, vibration andtemperature stimuli on an innervated skin surface of the operator 900.Because sensor assembly 100 produces broadband responses to slip andtexture that are similar to the signals available to the nervous systemfrom biological tactile receptors, it enables a particularly realistichaptic display as follows. The signals produced by vibration sensingsubsystem 205 and pressure sensing subsystem 210 can be combined andapplied more or less directly onto the skin of a human operator 800 witha haptic display 700 in order to produce realistic illusions of contact,slip and texture. This would be an advance over vibrotactile informationcurrently delivered from tactors in other applications which typicallystimulate at a single frequency with modulated amplitude.

FIGS. 7 and 8 illustrated that slip and texture vibrations are acombination of multiple frequency components and amplitudes. Presentingsome or all of the complete set of force, vibration and temperatureinformation sensed by biomimetic sensor assembly 100 to the skin of anoperator 900 via a multimodal haptic display 700 would produce morerealistic illusions of contact, slip and texture. For human userinteraction such as telerobotics and prosthetics, this would be expectedto improve haptic perception and dexterous manipulation of objects.

Measurement of Force Vectors for Tactile Feedback Control

Stabilizing a grip may be a function whose requirements and naturalstrategies are starting to be well understood. In a series of papers byRoland Johansson and coworkers, it has been shown that the gripstability may be affected by an object's size and shape, its mass andweight distribution, and by the coefficient of friction between thefingertips and surface of the object. They have also shown that thecentral nervous system usually may adjust the grip force so that thefriction force developed between the fingertips and the object surfacemay have a small margin over the external forces that would otherwisecause the object to slip. This strategy may energetically be efficientand suitable for manipulating delicate objects that might be crushed,but it demands continuous tactile sensing and adjustment of grip forcesaccording to the perceived properties of the gripped object.

Each finger's grip force may be adjusted independently based on thesensory information from that finger only and on the local conditions interms of weight distribution and friction. At least some of thisadjustment may occur so rapidly that it appears to be mediatedreflexively in the spinal cord rather than via the brain. This isimportant for prosthetic limbs because it suggests that tactileinformation can serve a useful function even if communication channelsto provide conscious perception of touch to the operator remainnonexistent or primitive, as they are now. Algorithms for the automaticadjustment of grip using biomimetic strategies are likely to be valuablealso in telerobotic and purely robotic manipulators.

When combined with sensing mechanisms of detecting normal and tangentialforces it is possible to use the proposed feature extraction 300 todetermine the static friction coefficient as outlined in the flowchartin FIG. 11. This can be accomplished by using an actuator 600 to delivera fixed amount of normal contact force and a second actuator 600 togradually increase tangential forces. As the tangential force isincreased eventually the sensor will begin to slip along the contactsurface. This moment of slip can be determined from the slip-detectionfeature extraction 300. Just before the moment of slip the ratio ofnormal to tangential contact forces can be used to calculate the staticfriction coefficient (μ) from the formula:

$\mu = {\frac{F_{tangential}}{F_{normal}}.}$This friction coefficient is unique to the coupling of the skin and theobject it is touching.

One method of applying the calculated static friction in grip control isoutlined in a method proposed by Puchhammer. In this method the requiredgrip force is determined from the tangential forces detected, thefriction coefficient (μ) and a safety factor from the following equation

$\;{F_{grip} = {\frac{F_{tangential}}{\mu} \times {S.F.}}}$However, without a suitable method of determining the static frictioncoefficient between a the gripping surface and an unknown object theassumed friction coefficient can be combined with an unnecessarily highsafety factor to maintain grasp on objects with a wide range of frictionproperties, which is undesirable for the handling of fragile objects.

A secondary method for controlling grip can be implemented by using theslip-detection derived from feature extraction 300 circuitry toconstantly check for slip and adjust this static coefficient when slipis detected as depicted in the flow-chart in FIG. 12. This active methodof reevaluating the static coefficient can reduce errors that may haveoccurred due to noise in sensing components and errors in evaluating thestatic friction coefficient. Because this strategy is more effective atmaintaining grip the safety factor used in grip force from the equationabove can be reduced which is advantageous for reducing the powerrequired in the gripping actuator.

An additional method of controlling grip would be to slowly reducecontact force while holding an object and make small increases in forcewhen a slip is detected as depicted in the flow-chart in FIG. 13. Thiswould allow for further improvements in power savings of the mechanicalactuator, as the applied grip force would not deviate far from theneeded grip force. This is particularly valuable if the actuator isproducing more grip force than is needed to maintain grasp. Byconstantly slowly reducing grip force and checking for slip the gripcontroller is able to continuously survey if it is possible to maintaingrasp of an object with a lower grip force yet maintain grasp when gripforce becomes critical, thus reducing power required in the grippingactuator. This also offers the additional advantage of not needing todetermine the static friction coefficient as well as not requiring anysensors to detect normal and shear forces as described in the methodspresented above.

Robust detection of the onset of slip introduces the possibility foradvanced biomimetic grip control algorithms. One proposed method forsignal processing to determine slip from feature extraction 300utilizing the acoustic fluid-pressure data as recorded by pressuresensor 10 is to use a set of band-pass filters and logic that check forthe characteristic signatures of slip, which are the existence of highfrequency fluid-pressure fluctuations and the relative absence oflow-frequency fluid-pressure fluctuations. In order to reduce falsepositives when using this approach it is desirable to ensure that thesecriteria are met and maintained for a short period of time, but notexcessively long as the object may slip from grasp.

To determine a suitable time for slip-detection confirmation it isuseful to compare with delays in biological reflexive grip-adjustmentswhich have been shown to occur after 60-80 ms (Johansson). Thisbiological reaction time is attributed to the transmission delays ofsensory and motor neurons (Kandel) as they travel from the fingertips tothe spinal cord and back as well as activation times in gripping muscles(Kandel). In comparison with an artificial grip-adjustment system neuraltransmission cables would be analogous to electric wires, which havevirtually non-existent transmission delays, and muscles would bereplaced with electromechanical actuators, which also have comparativelynegligible activation times. Therefore this 60-80 ms delay as found inthe biological system can be used to determine the confidence in slip asdetected using the aforementioned methods with no loss in performancewhen compared to the gold standard of biological reaction times for gripadjustments. Additional methods to accomplish this digitally would to beto use short-time Fourier transforms to analyze the frequency content.When converting to the frequency domain with a time window of this sizethe frequency resolution is also sufficient to distinguish betweenlow-frequency content and high-frequency content as describe above andrequired for accurate determination of slip. The ability to automate thedetection of slip would allow for advanced biomimetic grip controlmechanisms.

Using a sensor with normal and tangential force sensing mechanisms, canpermit for biomimetic exploratory movements such as stroking and rubbingtypically implemented for texture discrimination tasks. Controlling thecontact force and sliding velocity are also beneficial for extractinginformation to discriminate textures in this fashion. A flowchart forperforming such active exploratory movements and interpreting thevibration data thereby obtained in illustrated in FIG. 14.

With the proposed design an object can be located using robotic vision,proximity sensing, or through other means and a mechanical actuator 500with the sensor assembly 100 attached can be moved towards the objectuntil a contact is detected. Utilizing force and contact informationextracted from impedance data using feature extraction 300 commands canbe delivered to the controller 400 such that the desired contact forceand location are optimized in the sensor assembly 100. Once contactconditions are established the manipulator 500 and sensor assembly 100can be stroked along the surface while maintaining desired contactingforces and locations while recording spectral content from the pressuresensor 10. At the end of the movement spectral analysis can be performedon the recorded content and it can be compared to a library of materialsfor texture identification.

Thermal Sensing Temperature Compensation, Impedance-Based Force Sensing

As ionic fluids decrease in temperature, their conductivity tends todecrease. Without temperature compensation, this decrease inconductivity would result in an increase in sensed impedance, whichwould be interpreted as an increase in force applied between anelectrode 6 and the overlying skin 16. Referring to FIG. 1, this problemis overcome by including in core 1 a thermistor 15 with a sufficientlyshort time constant relative to temperature changes to be measured.

Thermal Characterization of Objects and Interactions Using Heat FlowSensing

If the sensor assembly 100 is heated, and contacted with an externalobject, its temperature will change appropriately with the mass,temperature, contact surface area, thermal conductivity and heatcapacity of the object. One method of heating the finger would bethrough the use of a heater. These tend to be bulky and would require acontroller for proper operation. Instead of using a heater, the powerdissipation energy from the electronics of the control board, 9, couldbe used to heat the finger. The heat dissipated by the microcontroller230 depends on its clock frequency and duty cycle of active use asopposed to “sleep states”. By utilizing a feedback signal from thethermistor 15, controller 400 can adjust the clock frequency oroperation duty cycle of microcontroller 230 to heat the finger to thedesired temperature. By keeping track of the amount of energy soapplied, it is possible to estimate the ambient temperature aroundsensor assembly 100 according to principles of thermodynamics that wouldbe obvious to one ordinarily skilled in the art.

When identifying an unknown object during an exploratory behavior, oneuseful piece of information concerns its thermal properties. Forexample, objects made of plastics, ceramics and metals tend to havedifferences in their thermal conductivity and heat capacity. Humansdetect such differences as they are reflected in the time-varying rateof temperature change of the skin when it comes into contact with anobject at a different temperature than that of the skin. The rate ofheat flow between the skin and object will depend on many factorsincluding the amount of pressure the object exerts on the sensorassembly 100. As the pressure increases, the amount of surface areaincreases, causing an increase in the rate of heat transfer. Thus,humans carefully control the location and amount of force of contactwith an object when attempting to extract information about its thermalproperties.

FIG. 15 illustrates an algorithm for determining thermal properties ofobjects based on the combination of temperature and force sensingcapabilities of exemplary embodiments of the present disclosure. Asensor assembly, e.g., sensor assembly 100 of FIG. 1, can be heated to acertain desired temperature above ambient by using feedback about thecurrent temperature of core from thermistor 15. To do this, controller400 adjusts the amount of power dissipated as heat by microcontroller230 by changing its clock frequency or duty cycle. In the event that themeasured temperature deviates from the desired value, a negativefeedback loop within the heat control software will compensate for thisto adjust the temperature back to the desired value. In order to obtaininformation about the thermal properties of an object, controller 400initiates an exploratory movement by sending commands to actuators 600.Information about the timing and nature of the contact with the objectis obtained from impedance sensing electrodes 6 in order to adjustactuators 600 so that the contact conforms to a standardized exploratorybehavior for determining thermal properties of objects. This behaviorwill generally require that the contact be centered over thermistor 15and that normal force of contact be sufficient to displace fluid inspace 2 allowing skin 16 to make firm contact directly with thermistor15.

Other opportunities to integrate information from various sensingmodalities described herein are intended to be covered by the scope ofthe present disclosure and would be obvious to someone normally skilledin the art. For example, with reference to FIG. 10, if the sensorassembly 100 is attached to a manipulator 500 capable of handling theobject, an estimation of object mass can be made from information thatis usually available to controller 400 from position, force, current orvoltage sensors related to powered actuators 600, much as humans do whenhaptically exploring objects. The thermal conductivity of the contactedobject can be estimated from the initial slope of the temperature changedetected by thermistor 15 and associated signal conditioning circuitry9. From the mass, thermal conductivity and the change in slope over timeof the temperature measured by thermistor 15, it is possible to estimatethe heat capacity of the material comprising the object. Thermalproperties of materials provide useful information about the identity ofthe materials and the identity of the object, particularly when combinedwith visual and other information commonly available from other sensingmodalities.

One skilled in the art will appreciate that embodiments and/or portionsof embodiments of the present disclosure can be implemented in/withcomputer-readable storage media (e.g., hardware, software, firmware, orany combinations of such), and can be distributed and/or practiced overone or more networks. Steps or operations (or portions of such) asdescribed herein, including processing functions to derive, learn, orcalculate formula and/or mathematical models utilized and/or produced bythe embodiments of the present disclosure, can be processed by one ormore suitable processors, e.g., central processing units (“CPUs)implementing suitable code/instructions in any suitable language(machine dependent on machine independent).

While certain embodiments have been described herein, it will beunderstood by one skilled in the art that the techniques (methods,systems, and/or algorithms) of the present disclosure may be embodied inother specific forms without departing from the spirit thereof.Accordingly, the embodiments described herein, and as claimed in theattached claims, are to be considered in all respects as illustrative ofthe present disclosure and not restrictive.

1. A biomimetic tactile sensor system for determining thermal properties of an object comprising: a rigid core having a surface; an elastomeric skin surrounding at least a portion of the core, having an inner and outer surface, and configured to form a space for confining fluid between the surface of the portion of the core and the inner surface of the elastomeric skin; a plurality of electrodes disposed on the surface of the rigid core and configured to be in contact with the fluid in a manner that causes the impedance between at least two of the electrodes to change as the amount of force between the outer surface of the elastomeric skin and the object changes when fluid is within the space between the surface of the portion of the core and the inner surface of the elastomeric skin; a temperature sensor configured and arranged to detect a temperature of the surface of the rigid core; a heater configured to heat the temperature sensor above ambient temperature; and an algorithm configured to determine thermal properties of the object based on the impedance between the at least two of the electrodes and the slope of temperature change detected by the temperature sensor.
 2. The system of claim 1, wherein the detection circuitry is configured and arranged to provide as an output a control signal for position and/or force based on the impedance.
 3. The system of claim 1, further comprising a fluid disposed within the space, wherein the fluid comprises an electrically conductive liquid.
 4. The system of claim 1, wherein the temperature sensor comprises a thermistor.
 5. The system of claim 1, wherein the heater includes a microprocessor configured and arranged to have power dissipation be controlled by software.
 6. The system of claim 1, wherein the temperature sensed by the temperature sensor is used to control the heater.
 7. A biomimetic tactile sensor system comprising: a rigid core having a surface and a plurality of electrodes disposed on the surface, wherein the electrodes are configured and arranged to detect impedance changes of a fluid; an elastomeric skin surrounding at least a portion of the core and having an inner and outer surface, and configured and arranged to form a space for confining a fluid between the surface of the core and the inner surface, wherein the inner surface of the skin comprises a plurality of asperities; and detection circuitry configured and arranged to detect changes in electrical impedance of the fluid between at least one of the electrodes caused by force that is exerted on the elastomeric skin, wherein the asperities are configured to cause the electrical impedance to increase in response to an increase in the force that is exerted on the elastomeric skin after the force reaches a level that causes the elastomeric skin to come in contact with the at least one electrode.
 8. The system of claim 7, wherein the detection circuitry is configured and arranged to provide as an output a control signal for controlling position and/or force based on the detected changes in impedance.
 9. The system of claim 7, further comprising a fluid disposed within the space, wherein the fluid comprises an electrically conductive liquid.
 10. The system of claim 7, wherein the asperities comprise a repeated pattern.
 11. The system of claim 7, wherein the asperities comprise pyramidal protrusions from the inner surface of the skin.
 12. The system of claim 7, wherein the asperities comprise columnar protrusions from the inner surface of the skin.
 13. The system of claim 7, wherein the asperities are cylindrical asperities.
 14. The system of claim 13, wherein the asperities are about 0.25 mm tall and 0.25 mm in diameter.
 15. The system of claim 14, wherein the asperities are about 0.5 mm apart and are configured and arranged in a regular square grid pattern.
 16. The sensor of claim 7, wherein the elastomeric skin comprises silicone or polyurethane.
 17. The sensor of claim 7, wherein the elastomeric skin includes bumps or ridges on the outer surface.
 18. A biomimetic tactile sensor system comprising: a rigid core having a surface; an elastomeric skin surrounding at least a portion of the core and having an inner and outer surface, and configured and arranged to form a space for confining a fluid between the surface of the core and the inner surface; one or more sensors configured and arranged to detect a physical characteristic of the fluid within the sensor system; a collar seal for sealing the skin against the core in a fluid tight configuration.
 19. The system of claim 18, further comprising a fluid disposed within the space, wherein the fluid comprises an electrically conductive liquid.
 20. The system of claim 18, further comprising an inflation port disposed within the skin.
 21. The system of claim 18, wherein the collar seal comprises a self-compressing edge seal that is molded as an integral part of the skin for simplifying removal and replacement of the skin for field service.
 22. The system of claim 21, wherein the collar seal further comprises a groove configured and arranged around a circumference of a portion of the core.
 23. A biomimetic tactile sensor system comprising: a rigid core having a surface; an elastomeric skin surrounding at least a portion of the core and having an inner and outer surface, and configured and arranged to form a space for confining a fluid between the surface of the core and the inner surface; one or more sensors configured and arranged to detect a physical characteristic of the fluid within the sensor system; and a resealable inflation port disposed within the skin for filling the space with fluid.
 24. The system of claim 23, further comprising conditioning circuitry configured and arranged to receive and condition electrical signals received from the one or more sensors.
 25. The system of claim 23, further comprising a fluid disposed within the space, wherein the fluid comprises an electrically conductive liquid.
 26. The system of claim 23, wherein the resealable inflation port comprises a plate with an aperture for receiving a needle.
 27. The system of claim 26, wherein the resealable inflation port comprises a thickened portion of the skin corresponding to the location of the plate, wherein when the plate is secured to the skin, compressive forces exist in the thickened portion.
 28. The system of claim 26, wherein either or both of the plate and core include one or more rigid surfaces that are angled for producing resultant forces within the thickened portion of skin producing compression perpendicular to an axis of a needle inserted through the skin for filling the space. 